Deep-Unfolding Neural-Network Aided Hybrid Beamforming Based on Symbol-Error Probability Minimization

In massive multiple-input multiple-output (MIMO) systems, hybrid analog-digital (AD) beamforming can be used to attain a high directional gain without requiring dedicated radio frequency (RF) chain for each antenna element, which substantially reduces both the hardware costs and power consumption. While MIMO transceiver design typically relies on conventional mean-square error (MSE) criterion, directly minimizing symbol rate (SER) lead superior performance. this article, we first mathematically formulate problem of under minimum SER (MSER) optimization criterion then develop an MSER-based iterative gradient descent (GD) algorithm find related stationary points. We propose deep-unfolding neural network (NN). The GD is unfolded into multi-layer structure wherein trainable parameters are introduced accelerate convergence enhance overall system To implement training stage, derive relationship between adjacent layers' gradients based generalized rule (GCR). NN developed quadrature phase shift keying (QPSK) $M$ -ary amplitude modulated (QAM) signals, its investigated theoretically. Furthermore, analyze transfer capability, computational complexity, generalization capability proposed NN. Our simulation results show that latter significantly outperforms counterpart at reduced complexity.